Papers with annotation pipeline
Bias Analysis and Mitigation through Protected Attribute Detection and Regard Classification (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models acquire general knowledge from pretraining but pretraining data contain undesirable social biases which can be perpetuated or even amplified by LLMs. |
| Approach: | They propose an efficient yet effective annotation pipeline to investigate social biases in pretraining data. |
| Outcome: | The proposed pipeline investigates social biases in the pretraining corpus using protected attribute detection and regard classification. |
TaeBench: Improving Quality of Toxic Adversarial Examples (2025.naacl-industry)
Copied to clipboard
| Challenge: | Existing adversarial examples generate invalid or ambiguous examples that fool the systems into wrong detection. |
| Approach: | They propose an annotation pipeline for quality control of generated toxic adversarial examples (TAE) they use model-based automated annotation and human-based quality verification to assess quality requirements of a TAE dataset. |
| Outcome: | The proposed pipeline can transfer-attack SOTA toxicity content moderation models and services with adversarial training. |
Efficient Annotator Reliability Assessment with EffiARA (2025.acl-demo)
Copied to clipboard
| Challenge: | Obtaining annotations from experts is ideal, but this expertise is logistically and financially costly. |
| Approach: | They propose an annotation framework that supports the whole annotation pipeline from understanding the resources required for an annotation task to compiling the annotated dataset. |
| Outcome: | The proposed framework improves classification performance through annotator-reliability-based soft-label aggregation and sample weighting, and increases agreement among annotators through removal of identifying and replacing an unreliable annotation. |
AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories (2024.findings-emnlp)
Copied to clipboard
Yifan Song, Weimin Xiong, Xiutian Zhao, Dawei Zhu, Wenhao Wu, Ke Wang, Cheng Li, Wei Peng, Sujian Li
| Challenge: | Existing studies focus on specialized agents designed for particular tasks. |
| Approach: | They propose to scale annotated interaction trajectories and fine-tune LLMs on AgentBank to get a series of agent models, Samoyed. |
| Outcome: | The proposed model can scale to get generalized agent capabilities. |
An Investigation of Noise in Morphological Inflection (2023.findings-acl)
Copied to clipboard
| Challenge: | Neural morphological inflection systems can be used for languages with very little supervised data, but are often less likely to have clean, goldstandard data. |
| Approach: | They propose an error taxonomy and annotation pipeline for inflection training data and propose a character-level masked language modeling (CMLM) pretraining objective. |
| Outcome: | The proposed pipeline is based on error taxonomy and annotation pipelines for unsupervised morphological paradigm completion. |
Emosical: An Emotion-Annotated Musical Theatre Dataset (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Emosical provides rich emotion annotations for musical films by inferring the background story of the characters. |
| Approach: | They propose to use a multimodal dataset of musical films to generate annotated emotion tags for each sample by inferring the background story of the characters. |
| Outcome: | The proposed dataset provides rich emotion annotations for musical films by inferring the background story of the characters. |
Curating a Large-Scale Motivational Interviewing Dataset Using Peer Support Forums (2022.coling-1)
Copied to clipboard
| Challenge: | Existing therapeutic chatbots lack large-scale conversations between clients and trained counselors . prior work has found that social media platforms such as Reddit are used to vent distress and peers are seen to actively respond to such posts. |
| Approach: | They propose to use peer support platforms to scrape conversational data from Reddit to determine whether counselors' responses align with real therapeutic conversations. |
| Outcome: | The proposed method achieved 97% coverage out of 17.3K responses, meaning that out of 16.8K responses labeled with a moderate agreement. |
Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models (2025.emnlp-main)
Copied to clipboard
Mehdi Ali, Manuel Brack, Max Lübbering, Elias Wendt, Abbas Goher Khan, Richard Rutmann, Alex Jude, Maurice Kraus, Alexander Arno Weber, Felix Stollenwerk, David Kaczér, Florian Mai, Lucie Flek, Rafet Sifa, Nicolas Flores-Herr, Joachim Koehler, Patrick Schramowski, Michael Fromm, Kristian Kersting
| Challenge: | Existing open-source multilingual datasets rely on heuristic filtering methods restricting both their cross-lingual transferability and scalability. |
| Approach: | They propose a systematic approach that curates diverse and high-quality multilingual data at scale while significantly reducing computational demands. |
| Outcome: | Evaluated empirically across 35 languages, the proposed approach outperforms current heuristic filtering methods like Fineweb2 and improves model training quality and retention rates. |
READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noises (2023.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for Chinese inputs often lack a realistic representation of real-world noises. |
| Approach: | They construct a Chinese multi-task benchmark with REalistic and Diverse input noises . they use pinyin input and speech input to recruit speakers from diverse dialects based on their inputs - a feature that is important for Chinese NLP benchmarks if it is implemented in real-world applications. |
| Outcome: | The proposed benchmarks are based on four different tasks and are designed to maximize diversity. |
PoS Tagging, Lemmatization and Dependency Parsing of West Frisian (2022.lrec-1)
Copied to clipboard
Wilbert Heeringa, Gosse Bouma, Martha Hofman, Jelle Brouwer, Eduard Drenth, Jan Wijffels, Hans Van de Velde
| Challenge: | a lemmatizer/PoS tagger/dependency parser for west frisian is released as a web app and as . web service. |
| Approach: | They propose a lemmatizer/PoS tagger/dependency parser for West Frisian using a corpus of 44,714 words in 3,126 sentences that were annotated according to the guidelines of Universal Dependencies version 2. |
| Outcome: | The proposed lemmatizer/PoS tagger/dependency parser performs better than the previous version of Oersetter . the current corpus contains 44,714 words in 3,126 sentences . |
STARS: A Unified Framework for Singing Transcription, Alignment, and Refined Style Annotation (2025.findings-acl)
Copied to clipboard
Wenxiang Guo, Yu Zhang, Changhao Pan, Zhiyuan Zhu, Ruiqi Li, ZheTao Chen, Wenhao Xu, Fei Wu, Zhou Zhao
| Challenge: | Existing automated singing annotation (ASA) methods tackle isolated aspects of the annotation pipeline. |
| Approach: | They propose a framework that addresses transcription, alignment, and refined style annotations. |
| Outcome: | The proposed framework delivers comprehensive multi-level annotations encompassing: (1) precise phoneme-audio alignment, (2) robust note transcription and temporal localization, (3) expressive vocal technique identification, and (4) global stylistic characterization including emotion and pace. |
Mapping Toxic Comments Across Demographics: A Dataset from German Public Broadcasting (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing toxic speech datasets lack demographic context and age data are limited . funk and its subsidiary accounts target users aged 14-29 . |
| Approach: | a german project introduces a large-scale toxic speech dataset annotated for toxicity . the dataset includes 3,024 human-annotated and 30,024 LLM-annnotated comments . researchers used human expertise and state-of-the-art language models to label comments based on toxic keywords . |
| Outcome: | The study combines human expertise with state-of-the-art language models to identify toxic speech categories. |